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1.  Introduction 


Statistical  software  packages,  to  large  extent,  accept  any  properly  configured 
data  set  and  proceed  to  process  it.  Few  if  any  checks  are  made  to  ensure  the  adequacy 
of  the  data  and  the  suitability  of  the  analysis,  and  little  is  done  to  provide  an 
explanation  or  interpretation  of  the  results.  This  requires  a  great  deal  from  the  user. 
Declining  computation  costs,  together  with  increased  availability  of  computers  and 
proliferation  of  statistical  software,  has  further  enhanced  the  opportunity  for  faulty 
data  analysis.  Application  of  expert  system  techniques  from  artificial  intelligence  to 
produce  more  cognizant  software  is  one  approach  to  reversing  this  unfortunate  trend. 

In  1985,  a  workshop  sponsored  by  AT&T  Bell  Laboratories  brought  together 
many  of  the  active  investigators  in  artificial  intelligence  and  statistics  and  was  the 
genesis  of  a  book  by  the  same  title  edited  by  Gale  [1].  This  reference  is  in  essence  the 
proceedings  of  the  workshop;  but  the  papers  given  there,  some  with  extensive 
bibliographies,  provide  the  most  complete  centrally-located  account  of  research  in  this 
topic  to  date. 

This  report  details  an  effort  underway  at  the  US  Army  Ballistic  Research 
Laboratory  (BRL)  to  develop  a  consultation  system  for  analysis  of  data  using 
nonparametric  statistical  procedures.  The  system,  called  Nonpare,  is  intended  to 
serve  a£  an  intelligent  interface  that  will  act  as  a  guide,  an  instructor,  and  an 
interpreter  to  a  body  of  statistical  software.  The  system  is  currently  a  prototype,  with 
a  first  release  planned  for  1989  for  field  testing. 

2.  Nonpare 

Nonparametric  statistics  is  too  large  an  area  to  hope  to  encompass  at  once, 
especially  if  the  entire  field  of  mathematical  statistics  is  partitioned  into  parametric 
and  nonparametric  procedures.  The  common-sense  approach  to  construction  of 
consultation  systems  suggests  limiting  the  domain  of  application,  but  nonparametric 
statistics  has  qualities  that  make  it  strongly  appealing. 

Nonparametric  data  analysis  is  characterized  chiefly  by  the  absence  of 
restrictive  distribution  assumptions -notably  freedom  from  dependence  on  the  normal 
(Gaussian)  distribution.  Many  nonparametric  statistical  procedures  are  exact  rather 
than  approximate  for  small  data  sets,  and  they  are  the  only  confirmatory  procedures 
which  can  be  used  to  analyze  data  collected  on  a  nominal  or  an  ordinal  scale  of 
measurement.  For  these  and  other  compelling  reasons  advanced,  for  example,  by 
Conover,  [2]  Hollander  and  Wolfe,  [3]  and  Lehmann,  [4]  nonparametric  procedures 
find  use  in  a  wide  variety  of  disciplines. 

2.1  The  System  Structure 

Nonpare  uses  Genie,  an  expert  system  shell  developed  at  the  BRL,  [5]  to 
provide  a  frame-based  production  system  with  forward  and  backward  inferencing  as 
well  as  an  explanation  facility  that  allows  the  user  to  interrogate  the  system -what 
hypotheses  are  being  entertained,  what  rules  are  being  verified,  what  facts  are  in 
evidence.  Genie  was  chosen  over  commercial  expert  system  shells  for  the  research 
and  development  of  Nonpare  because  of  its  accessibility  for  modification. 
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Nonpare,  shown  schematically  in  figure  1,  consists  of  three  subsystems  in 
addition  to  Genie. 


Genie 


Figure  1.  Nonpare  system  overview. 


The  system  dictionary  is  a  facility  whose  purpose  is  to  provide  on-line 
explanation  of  statistical  jargon  that  may  appear  during  the  interactive  dialog  between 
Nonpare  and  the  user.  Expert  domain  knowledge,  codified  in  English-like  rules, 
resides  in  the  knowledge  base.  Once  an  appropriate  procedure(s)  has  been  identified, 
the  data  are  analyzed  and  the  results  explained  by  the  nonparametric  data  analysis 
component.  Graphics  is  used  to  summarize  the  data  and  enhance  the  explanation.  In 
total,  the  user  is  led  within  system  limitations  to  an  appropriate  statistical  procedure 
through  an  interactive  process  in  which  the  user  is  questioned  and  can  in  turn  question 
the  consultation  system.  Nonpare  is  written  in  Interlisp-D  and  currently  runs  on 
Xerox  1100  Series  Lisp  machines. 

3.  An  Illustrative  Session 

Following  the  dictum  of  American  educator  John  Dewey  (1859*1952)  that 
"We  learn  by  doing,"  a  detailed  session  with  Nonpare  follows,  in  which  the  main 
system  features  are  illustrated. 

Example  3.1 

Suppose  that  a  ballistician  needs  to  assess  the  effectiveness  of  a  newly 
designed  kinetic  energy  penetrator  against  a  specific  armor  plate.  In  particular,  the 
experimenter  would  like  to  establish  whether  the  probability  of  perforation  exceeds 
.80,  a  level  already  attained  with  existing  technology.  Fourteen  rounds  are  fired,  and 
[perforation  and  [njonperforation  recorded  to  obtain:  n,  p,  p,  p,  n,  p,  p,  p,  p,  n,  p,  p, 
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p,  p.  Is  the  Pr{perforation}  >  .80  ? 

(A  diversion  here.  Searching  for  a  statistical  procedure  with  a  set  of  data 
already  collected  is  precisely  how  not  to  proceed.  The  purpose  for  collecting  the  data 
should  first  be  established,  and  then  the  statistical  tools  available  to  support  this 
purpose  determined.  Then  the  collection  and  analysis  of  data  can  proceed  in  an 
informed  manner.  Lamentably,  the  methodology-search  scenario  is  enacted  over  and 
over  again;  so  this  example  is  not  too  contrived.) 

It  should  be  apparent  from  the  onset  that  the  question  regarding 
Pr {perforation}  >  .80  can  never  be  answered  unequivocally  yes  or  no,  but  only  with 
some  degree  of  qualification. 

Nonpare  presently  has  nineteen  distinct  data  analysis  procedures  at  its 
disposal;  the  number  continues  to  increase.  No  assumptions  have  been  made  about 
their  frequency  of  use;  one  procedure  has  not  been  declared  most  likely  to  be 
exercised,  a  second  procedure  next  most  likely,  and  so  on,  since  the  base  of  potential 
users  is  so  broad.  For  the  user,  this  means  that  any  procedure  is  a  likely  starting  point, 
as  in  this  session,  the  dialog  of  which  begins  in  figure  2.  In  the  remainder  of  this 
section,  the  conventions  that  boldface  denotes  system  prompts  and  brackets  contain 
user  input  will  be  adopted.  An  occasional  system  response  may  be  italicized  but 
should  not  be  confusing  within  the  context  of  its  appearance. 
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Figure  2.  Beginning  dialog  with  Nonpare. 

The  session  begins  with  a  question  about  the  configuration  of  the  data. 

Do  you  have  a  sample  X,,  ....  Xn  ?  The  data,  n,  ....  p,  look  like  X,,  ...,  Xn;  respond 
[y]es. 

Are  you  interested  in  whether  the  data  conform  to  a  specified  distribution  ? 
Nonpare  is  investigating  a  possible  goodness-of-fit  situation.  A  statistician, 
anticipating  an  approach  to  this  problem,  might  find  a  [y]es  response  is  appropriate 
here.  A  nonstatistician,  for  whom  this  portion  of  the  system  is  designed,  and  who  is 
interested  in  whether  Pr{perforation}  >  .80,  should  respond  [n]o,  as  indicated. 

Are  you  interested  in  the  probability  of  occurrence  of  a  particular  category  or 
event  ?  [y]es.  The  user  is  interested  in  the  probability  of  occurrence  of  a  perforation. 
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Enter  the  name  of  the  category  of  interest,  [perforation].  Domain-dependent 
terminology  is  being  introduced. 

Are  the  n  trials  producing  the  values  Xr  ...,  Xn  independent  ?  Suppose  the  user  is 
unsure  of  the  technical  implications  of  the  term  "independent."  An  acceptable 
response  is  [What  is  independent]  -  as  shown  in  figure  3. 


Are  the  n  trials  producing  the  values  XI,  ....  Xn  independent?  what  is  in 
dependent 

Independence  relates  to  treeaom  from  external  influence  or  control ~  here, 
the  reference  is  to  measurements  (data.)  being  free  to  assume  values 
without  regard  to  other  measurements  that  may  be  maoe 

Are  the  n  trials  producing  the  values  XI,  .  ...  Xn  independent?  y 

fioes  each  trial  have  the  same  probability  p  of  producing  the  perforation 

?  y 

Are  you  interested  in  considering  whether  the  probability  of  occurrence 
of  the  perforation  equals  or  is  bounded  by  some  specified  value  p*? A 


Figure  3.  A  call  to  the  system  dictionary. 

Independent  relates  to  freedom  from  external  influence  or  control-  here,  the  reference 
is  to  measurements  (data)  being  free  to  assume  values  without  regard  to  other 
measurements  that  may  be  made. 

This  illustrates  a  dilemma  for  the  subject  area  specialist.  It  may  be  impossible  to 
rigorously  define  a  term  without  reliance  upon  other  terms  that  are  equally  obscure  to 
a  user  with  only  a  modest  statistical  background.  This  is  the  case  here,  where 
independence  is  bound  to  basic  concepts  of  probability  theory.  Nonpare’s  response 
conveys  the  notion,  but  regrettably  not  the  substance,  of  independence.  More  work  is 
needed  here.  For  now,  assume  the  experimenter  has  collected  a  set  of  independent 
data. 

Are  then  trials  producing  the  values  X,,  ...,  Xn  independent  ?  [y]es. 

Does  each  trial  have  the  same  probability  p  of  producing  the  perforation  ?  [y]es. 
Notice  that  Nonpare  is  now  using  language  the  user  provided,  when  it  talks  about 
probability  of  perforation. 

Are  you  interested  in  considering  whether  the  probability  of  occurrence  of  the 
perforation  equals  or  is  bounded  by  some  specified  value  p*  ?  [y]es.  The  user  is 
interested  in  the  inequality  Pr{perforation}  >  .80.  After  a  [y]es  response,  the  system 
suggests  a  possible  approach,  shown  in  figure  4. 


The  b  i  1'iGfil  i  a  1  test  15  all  appropriate  procedure.  To  execute  the 
binomial  test,  use  the  menu  to  complete  this  statement: 

I  am  interested  in  testing  the  null  hypothesis  that : The  probability 
of  occurrence  of  the  perforation 


Pick  One 


equals  some  value  p* 
does  not  exceed  p* 
is  at  least  p* 


Figure  4.  A  call  to  the  nonparametric  data  analysis  subsystem. 


The  menu  allows  the  user  to  select  either  a  two-sided  or  one-sided  test  of  hypothesis 
and  is  a  potential  source  of  error.  Beginning  statistics  students,  not  realizing  that  a 
null  (or  empty)  hypothesis  is  chosen  to  be  rejected,  might  mistakenly  choose  is  at  least 
p*  at  this  juncture.  Here  again,  some  level  of  statistical  competence  is  required. 
Selecting  the  hypothesis  does  not  exceed  p*  from  the  menu  using  a  mouse,  the  user 
obtains  for  confirmation  (figure  5)  the  statement: 

I  am  interested  in  testing  the  null  hypothesis  that:  The  probability  of  occurrence  of  the 
perforation  does  not  exceed  p*. 


I  am  interested  in  testing  the  null  hypothesis  that \  The  probability  of 
occurrence  of  the  perforation  does  not  exceed  p* 

Specify  the  sample  size  n  ->  14 
Specify  a  value  for  p*  ->  8G 

Specify  the  number  of  datum  values  assigned  to  the  perforation  ->  11 


Figure  5. 


Hypothesis  confirmation  and  input  parameter  declaration. 


Specify  the  sample  size  n.  [14] 
Specify  a  value  for  p#.  [.80] 
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Specify  the  number  of  datum  values  assigned  to  the  perforation.  [11] 

The  first  two  "Specify  ... "  commands  determine  the  appropriate  binomial  distribution; 
the  third  determines  the  size  of  the  critical  region  for  the  statistical  procedure,  which 
is  explained  in  figure  7,  following  the  system-generated  histogram  shown  in  figure  6. 


Region  of  Rejection 


X 


Figure  6.  Statistical  graph  summary. 

The  histogram  displays  the  probability  of  observing  exactly  n  (n=0,  ...,  14) 
armor  perforations  in  fourteen  shots  if  the  true  (but  unknown)  Pr{ perforation)  *  .80. 
A  statistician  will  readily  assimilate  this  graph.  If  the  user  merely  looks  at  it  as  a  plot 
involving  n  rounds  in  which  the  light  gray  region,  corresponding  to  n  >  11,  holds  some 
special  significance,  and  it  provides  some  reassurance  regarding  the  unseen 
computations,  it  will  have  served  its  purpose  here.  Figure  7,  which  appears  on  the 
terminal  simultaneously,  explains  that 

The  critical  level  of  this  test,  corresponding  to  the  light  gray  region,  is  .69 
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This  means  that  if  you  reject  the  hypothesis  ( The  probability  of  occurrence  of  the 
perforation  does  not  exceed  .8)  you  do  so  with  a  .69  probability  of  being  in  error. 


The  critical  level  of 
is  .69 

this  test,  com  e 

spending  to 
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This  means  that,  if  you 
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C  ed'.r  '.!a  i  ’ 

Figure  7.  Explanation  and  interpretation  of  results. 

Since  the  investigation  began  with  the  assumption  (null  hypothesis)  that  the 
Pr{perforation}  <  .80,  the  evidence  collected -eleven  perforations,  three 

nonperforations  -  is  not  sufficient  to  support  abandonment  of  that  assumption.  A 
probability  of  being  in  error  of  .69  is  more  than  a  reasonable  person  would  be  willing 
to  assume.  And  so,  the  response  to  the  original  question.  Is  the  Pr{perforation)  > 
.80  ?  is  a  qualified  no,  the  qualification  being  expressed  through  invocation  of  the 
critical  level. 

Would  you  like  to  run  the  binomial  procedure  again  ? 

At  this  juncture,  an  experimenter  might  well  be  asking  a  number  of  "What  if  ..." 
questions.  "What  if  I  had  been  able  to  afford  three  more  firings?"  or,  "What  if  I  had 
observed  one  more  perforation?"  and  so  on.  A  response  of  [y]es  here  allows  the  user 
to  exercise  the  binomial  procedure  directly,  without  having  to  respond  again  to  all  the 
preliminary  questions.  A  [n]o  response  is  given,  but  this  is  an  excellent  place  to  use 
Nonpare’s  tutorial  capabilities  to  study  the  sensitivity  of  the  binomial  procedure  to 
modification  of  parameter  values  or  slight  changes  in  the  data. 

Are  you  interested  in  determining  an  interval  in  which  the  probability  p  of 
occurrence  of  the  perforation  lies? 

The  foregoing  analysis  suggests  that  an  assertion  that  the  probability  of  perforation 
lies  in  the  interval  (.80,  1]  cannot  be  made.  What  interval  might  be  expected  to 
capture  this  unknown  parameter?  A  response  of  [y]es  causes  this  question  to  be 
answered,  first  graphically,  as  in  figure  8,  and  then  verbally,  as  in  figure  9. 
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CONFIDENCE  INTERVAL 

wilh  95.0%  Confidence  Level 


Figure  8.  Display  for  a  95%  confidence  interval. 

Figure  8  shows  that  the  Pr{perforation},  whose  estimate  based  on  the 
fourteen  firings  is  p  =  .78,  lies  within  the  interval  [.48,  .94]  with  a  high  level  of 
confidence.  This  interval  is  so  broad  one  can  see  why  the  assertion  that 
Pr{ perforation}  >  .80  is  ill-advised.  The  formal  interpretation  of  the  confidence 
interval  is  given  as 

The  probability  of  occurrence  of  the  perforation  is  contained  in  the  interval  [.48, 
.94]  with  an  a  priori  probability  .95  . 


Are  you  interested  in  determining  an  interval  in  which  the  probability  p 
of  occurrence  of  the  perforation  lies?  y 


The  probability  of  occurrence  of  the  perforation  is  contained  in  the 
interval  [.48,. 94]  with  an  a  priori  probability  .95. 

Would  you  like  a  confidence  level  other  than  ,95  ?  n 


Figure  9.  Explanation  and  interpretation  of  the  confidence  interval. 

Would  you  like  a  confidence  level  other  than  .95  ?  [no].  The  95%  confidence  level 
was  prechosen.  A  [yjes  response  allows  the  user  to  control  the  confidence  level.  The 
session  is  terminated  with  a  [n]o  response,  shown  in  figure  9. 

At  the  conclusion  of  the  session  the  inference  engine  displays  a  fact  solution 
tree  for  all  the  intermediate  decisions  leading  to  the  final  conclusion.  Buttoning  with  a 
mouse  any  node  of  the  fact  tree  produces  the  logic  leading  to  that  location.  In  figure 
10,  factll  was  buttoned,  and  the  corresponding  trace  is  displayed  beneath  the  fact 
tree.  These  are  features  of  the  inference  engine  rather  than  Nonpare,  but  they  are 
valuable  as  diagnostics  to  the  developer  and  provide  some  measure  of  reassurance  to 
the  user. 


Figure  10.  Session  conclusion 


4.  Conclusions 

Nonpare,  a  consultation  system  for  analysis  of  data  using  nonpar ametric 
statistical  procedures,  has  been  described;  and  most  of  its  operational  features  have 
been  illustrated.  The  essence  of  the  system  is  the  rule-based  interface  with 
accompanying  software  for  data  analysis  and  the  interpretation  of  the  ensuing 
computations.  Nonpare  is  under  active  development,  but  its  feasibility  as  an 
operational  system  has  been  established  Enlargement  of  the  rule-base  and  the 
addition  of  more  statistical  procedures  is  clearly  indicated  before  it  can  approach  its 
potential.  Not  surprisingly,  tangential  problems  in  basic  research  have  been  spawned 
by  this  effort.  A  first  release  is  planned  for  1989  for  field  testing. 
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